Advancements in computing and communications technology have made it possible to develop computerized models that are digital simulations of real life systems or structures (i.e., physical models). For example, using sensors to monitor different aspects of a bridge, an industrial plant or a medical monitoring system, a variety of data may be captured about different elements and dynamics of such systems and how related elements interact or affect other internal and external relationships.
Depending on implementation, data may be captured over a dedicated course of time or the lifespan of a target system by way of processing signals generated by sensors positioned, connected or otherwise associated with various parts of the system. Using such monitoring technology, it is possible to collect vast amounts of data about the target system. The collected sensor data or signals can be in turn utilized for the purpose of simulation, detecting and preventing faults or to better understand certain system operations and functionalities.
Sensor data is typically stored and provided as input to simulation applications. The data can be stored or may originate from different data sources, such as files in a file system. Data from different data sources is usually formatted differently. Further, depending on the data source, data packaging and transmission may be provided in different payload sizes and time series. Even further, communication channels utilized for delivery and data transmission may be configured according to different sequencing or configuration requirements.
For the data to be useful as input to an application, the above-noted variety of factors associated with data formatting, sourcing and delivery need to be managed by way of reliance on expertise in systems integration and engineering, so that a receiving application can successfully process the input data and generate viable output data. Typically, systems integration requires the expertise of system engineers or software developers who create custom code or programs that configure data from a designated source for input into an application, and further adapt the application's output for a specific destination or resource.
The effort needed to fully integrate and harmonize input or output data received or generated by a large number of applications (e.g., in a cloud-based environment) with various input and output mechanisms may be very time consuming and complex. For example, due to a lack of standardization and depending on the needs of an application or user, input or output data may be provided or stored in various data formats in association with different sources or destinations, for example, in a local file system, onto cloud storage, or by way of buffering data streams.
Due to the difficulty and overhead associated with processing data originated from different sources with different formatting and packaging requirements, the cost and effort for developing applications that can successfully handle such mix of formatting and configurations is very high. For the same reasons, it is generally infeasible to test a prototype application across different computing architectures, using real-time and large-scale data platforms, particularly in a cloud-based environment. As such, enhanced computing systems and technological improvements are needed that can help overcome the above shortcomings.